Papers with key-value (KV)
Adaptive Layer Selection for Layer-Wise Token Pruning in LLM Inference (2026.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated remarkable capabilities in processing long contexts. |
| Approach: | They propose a training-free method that adaptively chooses the selection layer for KV cache reduction . they exploit the variance of token ranks ordered by attention score to optimize decoding . |
| Outcome: | The proposed method outperforms state-of-the-art token pruning methods on InfiniteBench, RULER, and NIAH benchmarks. |
EvolKV: Evolutionary KV Cache Compression for LLM Inference (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing key-value (KV) cache compression methods ignore interplays between layer-specific feature patterns and task performance. |
| Approach: | They propose an adaptive framework for layer-wise, task-driven KV cache compression that optimizes memory efficiency and task performance. |
| Outcome: | EvolKV outperforms baseline methods on long-context tasks and surpasses heuristics by 7 percentage points on GSM8K. |
TaDA: Training-free recipe for Decoding with Adaptive KV Cache Compression and Mean-centering (2025.acl-industry)
Copied to clipboard
| Challenge: | key-value caches in large language models consume memory, posing a major challenge for scalable deployment. |
| Approach: | They propose a training-free recipe for KV cache compression with quantization precision that adapts to error sensitivity across layers and a mean centering to eliminate separate outlier handling. |
| Outcome: | The proposed technique reduces the KV cache memory footprint to 27% of the original 16-bit baseline while achieving comparable accuracy. |
InfiniSST: Simultaneous Translation of Unbounded Speech with Large Language Model (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing models for simultaneous speech translation assume pre-segmented speech, limiting their real-world applicability. |
| Approach: | They propose a multi-turn dialogue task that can translate unbounded streaming speech . they construct translation trajectories and robust segments from MuST-C with multi-latency augmentation during training and develop a cache management strategy to facilitate efficient inference. |
| Outcome: | The proposed approach reduces computation-aware latency by 0.5 to 1 second while maintaining the same translation quality compared to baselines. |
OjaKV: Context-Aware Online Low-Rank KV Cache Compression (2026.findings-acl)
Copied to clipboard
Yuxuan Zhu, David H. Yang, Mohammad Mohammadi Amiri, Keerthiram Murugesan, Tejaswini Pedapati, Pin-Yu Chen
| Challenge: | Existing methods for inference use static, offline-learned subspaces that perform poorly under distribution shifts. |
| Approach: | They propose a framework that integrates a storage policy with an online subspace adaptation to preserve key-value tokens in full rank as high-fidelity anchors. |
| Outcome: | Experiments show that OjaKV maintains or improves zero-shot accuracy at high compression ratios, achieving the strongest gains on long-context benchmarks requiring complex reasoning. |
Layer-Condensed KV Cache for Efficient Inference of Large Language Models (2024.acl-long)
Copied to clipboard
| Challenge: | Using a key-value cache, memory consumption is a bottleneck for high-throughput language models. |
| Approach: | They propose a method that only computes and caches the KVs of a small number of layers, thus saving memory consumption and improving inference throughput. |
| Outcome: | The proposed method achieves higher throughput and competitive performance than standard transformers and is orthogonal to existing transformer memory-saving techniques. |